Skip to content

acc: run ssh/connect-serverless-gpu locally#5878

Merged
radakam merged 4 commits into
mainfrom
acc-local-ssh-connect-serverless-gpu
Jul 13, 2026
Merged

acc: run ssh/connect-serverless-gpu locally#5878
radakam merged 4 commits into
mainfrom
acc-local-ssh-connect-serverless-gpu

Conversation

@radakam

@radakam radakam commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Changes

Local test coverage for databricks ssh connect, which was disabled on cloud and local in #4838, across the two layers we can exercise without compute:

  1. Bootstrap job submissionacceptance/ssh/connect-serverless-gpu runs against the in-process testserver and asserts the serverless-GPU job the CLI submits (GPU_1xA10, serverless env, secret scope, notebook task). This needs new libs/testserver stubs: secrets/listRESOURCE_DOES_NOT_EXIST for a missing scope, a synthesized metadata.json on ssh-server-bootstrap-* submit, driver-proxy metadata/logs, and workspace/export404 for a missing object.
  2. Real SSH handshake through the proxyclient_server_sshd_test.go stands up a real sshd behind proxy.NewProxyServer and drives a real SSH client (x/crypto/ssh) through RunClientProxy. The test-exp-ssh job installs openssh-server on Linux to support it.

Why

ssh connect currently has no automated coverage. These tests exercise the two layers reachable locally: the control-plane setup (the bootstrap job the CLI submits) and the tunnel transport carrying the real SSH protocol.

A full local handshake via the connect command can't complete (no compute, a wss:// driver-proxy dial, real ssh spawn), so the acceptance test stops at the connection attempt — failure output goes to LOG.stderr (excluded from goldens), and the cloud Connection successful path is preserved for when cloud is re-enabled. The handshake test is Linux-only (//go:build linux): running sshd -i as non-root is reliable there but not on macOS; it skips when sshd is absent. The end-to-end path through the real driver proxy (bootstrap notebook → databricks ssh server on compute) still needs an integration test and is out of scope.

Tests

  • acceptance/ssh/connect-serverless-gpu (asserts the submitted bootstrap job payload)
  • TestClientServerRealSSHD in experimental/ssh/internal/proxy (real SSH handshake, pubkey auth, and remote command over the proxy tunnel), running on Linux in the test-exp-ssh job

Flip acceptance/ssh/connect-serverless-gpu to Local = true so it runs
against the in-process testserver (it was disabled on both cloud and
local in #4838). A full SSH handshake can't complete locally (no real
compute/sshd + a wss driver-proxy), so the local run drives the connect
flow up to the connection attempt and asserts the serverless GPU
bootstrap job the CLI submits (accelerator, environment, secret scope).

Teach libs/testserver enough of the SSH tunnel backend for the flow to
get there deterministically and fast:
- secrets/list (ListSecretsByScope), returning RESOURCE_DOES_NOT_EXIST
  for a missing scope so CreateKeysSecretScope creates it
- synthesize metadata.json on the ssh-server-bootstrap run submit, as a
  real tunnel server would publish it, so getServerMetadata succeeds
- driver-proxy metadata/logs endpoints
- workspace/export now 404s a missing object instead of returning a nil
  body (which the response normalizer rejects)

The connect attempt itself is expected to fail locally; its noisy
output goes to LOG.stderr (excluded from the golden files).
@radakam radakam temporarily deployed to test-trigger-is July 9, 2026 13:27 — with GitHub Actions Inactive
@radakam radakam temporarily deployed to test-trigger-is July 9, 2026 13:27 — with GitHub Actions Inactive
@radakam radakam temporarily deployed to test-trigger-is July 9, 2026 13:31 — with GitHub Actions Inactive
@radakam radakam temporarily deployed to test-trigger-is July 9, 2026 13:31 — with GitHub Actions Inactive
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 102a096

Run: 29233660916

Env 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 4 4 230 1078 4:55
💚​ aws windows 4 4 232 1076 6:52
🔄​ aws-ucws linux 3 3 4 314 995 5:49
💚​ aws-ucws windows 4 4 318 993 6:59
💚​ azure linux 4 4 230 1077 4:55
💚​ azure windows 4 4 232 1075 7:09
💚​ azure-ucws linux 4 4 318 992 5:43
💚​ azure-ucws windows 4 4 320 990 7:29
💚​ gcp linux 4 4 229 1079 5:07
💚​ gcp windows 4 4 231 1077 6:52
10 interesting tests: 4 SKIP, 3 flaky, 3 RECOVERED
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🔄​ TestAccept 💚​R 💚​R 🔄​f 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestAccept/ssh/connect-serverless-cpu 🙈​s 🙈​s 🔄​f ✅​p 🙈​s 🙈​s ✅​p ✅​p 🙈​s 🙈​s
🔄​ TestAccept/ssh/connect-serverless-cpu/DATABRICKS_BUNDLE_ENGINE=direct 🔄​f ✅​p ✅​p ✅​p
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 9 slowest tests (at least 2 minutes):
duration env testname
6:04 azure windows TestAccept
6:02 azure-ucws windows TestAccept
5:59 aws windows TestAccept
5:58 gcp windows TestAccept
5:53 aws-ucws windows TestAccept
2:57 aws linux TestAccept
2:54 gcp linux TestAccept
2:54 azure linux TestAccept
2:47 azure-ucws linux TestAccept

Add a proxy-level test that runs a real OpenSSH server (sshd -i) behind the
websocket proxy and a real SSH client through RunClientProxy, exercising the
actual handshake, pubkey auth, and remote command exec end-to-end without any
Databricks compute. This complements the existing `cat`-based transport tests
by validating that the tunnel carries the SSH protocol faithfully, giving us
local coverage of the connectivity path that otherwise needs a cluster.

The test is skipped when sshd (openssh-server) is not installed and is gated to
non-Windows, matching the existing proxy tests.
@radakam radakam temporarily deployed to test-trigger-is July 13, 2026 07:28 — with GitHub Actions Inactive
@radakam radakam temporarily deployed to test-trigger-is July 13, 2026 07:28 — with GitHub Actions Inactive
The runner images ship only openssh-client, so the real SSH handshake test
would silently skip on Linux (no sshd) and is unreliable to run as a non-root
user on macOS. Restrict it to Linux and install openssh-server in the
test-exp-ssh job so it runs deterministically there; it still skips gracefully
when sshd is absent (e.g. the general test job, which doesn't install it).
@radakam radakam temporarily deployed to test-trigger-is July 13, 2026 07:55 — with GitHub Actions Inactive
@radakam radakam temporarily deployed to test-trigger-is July 13, 2026 07:55 — with GitHub Actions Inactive
@radakam radakam marked this pull request as ready for review July 13, 2026 08:42
@radakam radakam requested a review from anton-107 July 13, 2026 08:42

@anton-107 anton-107 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but let's try and converge these tests in two different styles into one (acceptance) in one of the follow-up PRs

@radakam radakam added this pull request to the merge queue Jul 13, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 13, 2026
@radakam radakam added this pull request to the merge queue Jul 13, 2026
Merged via the queue into main with commit 7875f5b Jul 13, 2026
30 checks passed
@radakam radakam deleted the acc-local-ssh-connect-serverless-gpu branch July 13, 2026 13:10
radakam added a commit that referenced this pull request Jul 14, 2026
Apply the same convergence to the serverless-GPU acceptance test (follow-up to
#5878, where the review asked to fold the two test styles into one acceptance
test). Locally it now runs a single `ssh connect --proxy` that submits the same
serverless-GPU bootstrap job and pipes stdin/stdout over the driver-proxy
websocket echo route, asserting both the job payload and the echoed bytes.
ronaldz-db pushed a commit to ronaldz-db/cli that referenced this pull request Jul 15, 2026
## Changes

The dedicated-cluster counterpart to
`acceptance/ssh/connect-serverless-gpu` (databricks#5878). Flips
`acceptance/ssh/connection` to `Local = true` and asserts the bootstrap
job the CLI submits for `ssh connect --cluster` (`existing_cluster_id`,
secret scope, notebook task, `serverless=false`), reusing the testserver
SSH-tunnel stubs from databricks#5878.

The default test cluster (`TestDefaultClusterId`) is now a running
dedicated single-user cluster — the shape `ssh connect --cluster`
requires (`ValidateClusterAccess` rejects anything else), matching the
cloud `TEST_DEFAULT_CLUSTER_ID` it stands in for. Only
`bundle/deployment/bind/cluster` (filters to `cluster_name`) and the
`cmd/api` route stubs reference it, so no other goldens change.

## Tests

- `acceptance/ssh/connection` (asserts the submitted dedicated-cluster
bootstrap job payload)
radakam added a commit that referenced this pull request Jul 15, 2026
#5920)

## Changes

Follow-up to #5878, converging the SSH tunnel's two test styles into one
acceptance test. Two enabling changes: derive the driver-proxy websocket
scheme from the host (`http` → `ws`, else `wss`) instead of hardcoding
`wss` so the client can dial the plaintext local test server, and add a
testserver `/ssh` websocket route (via a new connection-hijacking
`Router.HandleRaw`) that upgrades the driver-proxy request to a **real
`sshd -i`** and bridges binary frames to its stdio, standing in for the
tunnel server a real cluster runs — no Databricks compute required.

The `/ssh` handler reads the client public key the CLI uploaded via
`secrets/put` (the ws dial carries the bearer token, so the token-scoped
workspace is the one that stored it), writes an ephemeral host key +
`authorized_keys` + a minimal `sshd_config`, and pumps websocket frames
to and from `sshd`'s stdin/stdout. `/metadata` now returns the current
OS user so `ssh -l <user>` matches the account `sshd` runs as.

`acceptance/ssh/connect-serverless-gpu` now runs a single local `ssh
connect ... -- "echo 'Connection successful'"` that both submits the
serverless-GPU bootstrap job and completes a genuine SSH handshake,
public-key auth, and remote command exec over the `ws://` tunnel,
asserting both the job payload and the command output. This makes
`TestClientServerRealSSHD` redundant, so it's removed; the `cat`-echo
transport tests in `client_server_test.go` remain.

Note the converged test exercises the real handshake on the **client
side only**: the `/ssh` route is a testserver stand-in, not the
production `proxy.NewProxyServer`/`runServerProxy`, so the server
proxy's transport stays covered by the `cat`-echo tests rather than by a
real handshake. This is a deliberate trade-off (`runServerProxy` is
protocol-agnostic), not an accidental coverage drop.

The handshake needs a real OpenSSH server: the test self-skips via
`SKIP_TEST` when `sshd` is absent (the general `test` job doesn't
provision it; `task test-exp-ssh` installs `openssh-server` on Linux).
It's pinned to Linux (`darwin`/`windows` disabled), where running `sshd
-i` as a non-root user with a throwaway config is reliable — matching
the removed Go test's rationale.

## Tests

* `acceptance/ssh/connect-serverless-gpu` (bootstrap job payload + real
SSH handshake and remote exec over the `ws://` tunnel)
* `TestBuildProxyWebsocketURL` (`ws`/`wss` scheme selection)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants